Inserting large volumes of data into SQLite - speed issues

by Marco Nelissen » Thu, 03 Sep 2009 01:02:38 GMT


Sponsored Links
 Are you using bulkInsert?





--~--~---------~--~----~------------~-------~--~----~



Inserting large volumes of data into SQLite - speed issues

by Gavin Aiken » Thu, 03 Sep 2009 01:10:08 GMT


 Does the data NEED to be on the device? Can it not reside on a server?
Certainly you are better to manage the synchonisation yourself rather than
bundle the 120MB lib for downloading through the marketplace. This sort of
download needs to be resumable and I would recommend users only download
with a Wifi connection.

What sort of data is this?! It would have to be incredible to justify that
size to most users, I think with a 3 hour install you have lost 99% of the
user base without even knowing if the app is any good.

Gav





--~--~---------~--~----~------------~-------~--~----~


Sponsored Links


Other Threads

1. Failed basic authentication with httpclient - 401 status code

I'm trying to port an open source RETS client to the Android platform.
The original client used Apache httpclient. I'm trying to test login
to a demo RETS server that uses basic authentication. The demo login
works fine when I test from web browsers. When I try a login from the
emulator I get a 401 status code. The login URL is
http://demo.crt.realtors.org:6103/rets/login. The user name is "Joe"
and the password is "Schmoe". I pasted my modified client code.

Grant

public class CommonsHttpClient extends RetsHttpClient {
    private static final Logger LOG =
Logger.getLogger(CommonsHttpClient.class.toString());

    private static final int DEFAULT_TIMEOUT = 300000;
    private static final String RETS_VERSION = "RETS-Version";
    private static final String RETS_SESSION_ID = "RETS-Session-ID";
    private static final String RETS_REQUEST_ID = "RETS-Request-ID";
    private static final String USER_AGENT = "User-Agent";
    private static final String RETS_UA_AUTH_HEADER = "RETS-UA-
Authorization";
    private static final String ACCEPT_ENCODING = "Accept-Encoding";
    public static final String CONTENT_ENCODING = "Content-Encoding";
    public static final String DEFLATE_ENCODINGS = "gzip,deflate";

    private final ConcurrentHashMap<String, String> defaultHeaders;
    private final DefaultHttpClient httpClient;

    // method choice improvement
    private final String userAgentPassword;

    public CommonsHttpClient() {
        this(DEFAULT_TIMEOUT, null, true);
    }

    public CommonsHttpClient(int timeout, String userAgentPassword,
boolean gzip) {
        this.defaultHeaders = new ConcurrentHashMap<String, String>();
        this.userAgentPassword = userAgentPassword;

        // setup the HTTP parameters
        HttpParams httpParams = new BasicHttpParams();
        HttpConnectionParams.setConnectionTimeout(httpParams,
timeout);
        HttpConnectionParams.setSoTimeout(httpParams, timeout);
        HttpProtocolParams.setVersion(httpParams,
HttpVersion.HTTP_1_1);
        // set to rfc 2109 as it puts the ASP (IIS) cookie _FIRST_,
this is critical for interealty
        HttpClientParams.setCookiePolicy(httpParams,
CookiePolicy.RFC_2109);

        // create a scheme registry for HTTP and HTTPS
        SchemeRegistry registry = new SchemeRegistry();
        registry.register(new Scheme("http", new PlainSocketFactory(),
80));
        registry.register(new Scheme("https",
SSLSocketFactory.getSocketFactory(), 443));

        // create a thread-safe connection manager. Allows multiple
threaded requests from client
        ThreadSafeClientConnManager multiThreadedConnectionManager =
                new ThreadSafeClientConnManager(httpParams, registry);
        this.httpClient = new
DefaultHttpClient(multiThreadedConnectionManager, httpParams);

        // we only support basic authentication
        BasicScheme basicScheme = new BasicScheme();
        AuthSchemeRegistry authRegistry = new AuthSchemeRegistry();
        authRegistry.register(basicScheme.getSchemeName(), new
BasicSchemeFactory());
        this.httpClient.setAuthSchemes(authRegistry);
        this.httpClient.setCredentialsProvider(new
BasicCredentialsProvider());

        // ask the server if we can use gzip
        if (gzip) this.addDefaultHeader(ACCEPT_ENCODING,
DEFLATE_ENCODINGS);
    }

    public HttpClient getHttpClient() {
        return this.httpClient;
    }

    //----------------------method implementations
    @Override
    public void setUserCredentials(String userName, String password) {
        this.httpClient.getCredentialsProvider().setCredentials(
                new AuthScope(AuthScope.ANY_HOST,
                        AuthScope.ANY_PORT,
                        AuthScope.ANY_REALM,
                        AuthScope.ANY_SCHEME),
                new UsernamePasswordCredentials(userName, password));
    }

    @Override
    public RetsHttpResponse doRequest(String httpMethod,
RetsHttpRequest request)
            throws RetsException {
        return "GET".equals(StringUtils.upperCase(httpMethod)) ?
this.doGet(request) : this.doPost(request);
    }

    //----------------------method implementations
    public RetsHttpResponse doGet(RetsHttpRequest request) throws
RetsException {
        String url = request.getUrl();
        String args = request.getHttpParameters();
        if (args != null) {
            url = url + "?" + args;
        }
        HttpGet method = new HttpGet(url);
        LOG.fine(String.format("\n>>>\nGET %s", url));
        return execute(method, request.getHeaders());
    }

    public RetsHttpResponse doPost(RetsHttpRequest request)
            throws RetsException {
        try {
            String url = request.getUrl();
            String body = request.getHttpParameters();
            if (body == null) body = "";  // commons-httpclient 3.0
refuses to accept null entity (body)
            HttpPost method = new HttpPost(url);
            method.setEntity(new StringEntity(body));
            method.setHeader("Content-Type", "application/x-www-url-
encoded");
            LOG.fine(String.format("\n>>>\nPOST %s\nArgs: %s", url,
body));

            return execute(method, request.getHeaders());
        } catch (UnsupportedEncodingException e) {
            throw new RetsException(e.getMessage());
        }
    }

    private RetsHttpResponse execute(final HttpUriRequest uriRequest,
Map<String, String> headers)
            throws RetsException {
        try {
            // add the default headers
            if (this.defaultHeaders != null) {
                for (Map.Entry<String, String> entry :
this.defaultHeaders.entrySet()) {
                    uriRequest.setHeader(entry.getKey(),
entry.getValue());
                }
            }
            // add our request headers from rets
            if (headers != null) {
                for (Map.Entry<String, String> entry :
headers.entrySet()) {
                    uriRequest.setHeader(entry.getKey(),
entry.getValue());
                }
            }

            final Map<String, String> cookies = this.getCookies();

            // optional ua-auth stuff here
            if (this.userAgentPassword != null) {
                uriRequest.setHeader(RETS_UA_AUTH_HEADER,
                        this.calculateUaAuthHeader(uriRequest,
cookies));
            }
            // try to execute the request
            final HttpResponse httpResponse =
this.httpClient.execute(uriRequest);
            if (httpResponse.getStatusLine().getStatusCode() !=
HttpStatus.SC_OK) {
                throw new InvalidHttpStatusException(
                        httpResponse.getStatusLine().getStatusCode(),
 
httpResponse.getStatusLine().getReasonPhrase());
            }

            return new CommonsHttpClientResponse(httpResponse,
cookies);
        } catch (Exception e) {
            throw new RetsException(e);
        }
    }

    @Override
    public synchronized void addDefaultHeader(String key, String
value) {
        this.defaultHeaders.put(key, value);
        if (value == null) this.defaultHeaders.remove(key);
    }

    private Map<String, String> getCookies() {
        Map<String, String> cookieMap = new CaseInsensitiveTreeMap();
        for (Cookie cookie :
this.httpClient.getCookieStore().getCookies()) {
            cookieMap.put(cookie.getName(), cookie.getValue());
        }
        return cookieMap;
    }

    private String calculateUaAuthHeader(HttpUriRequest uriRequest,
Map<String, String> cookies) {
        final String userAgent = this.getHeaderValue(uriRequest,
USER_AGENT);
        final String requestId = this.getHeaderValue(uriRequest,
RETS_REQUEST_ID);
        final String sessionId = cookies.get(RETS_SESSION_ID);
        final String retsVersion = this.getHeaderValue(uriRequest,
RETS_VERSION);
        String secretHash = DigestUtils.md5Hex(String.format("%s:%s",
userAgent, this.userAgentPassword));
        String pieces = String.format("%s:%s:%s:%s", secretHash,
                StringUtils.trimToEmpty(requestId),
                StringUtils.trimToEmpty(sessionId),
                retsVersion);
        return String.format("Digest %s", DigestUtils.md5Hex(pieces));
    }

    private String getHeaderValue(HttpUriRequest uriRequest, String
key) {
        Header requestHeader = uriRequest.getHeaders(key)[0];
        if (requestHeader == null) return null;
        return requestHeader.getValue();
    }
}


--~--~---------~--~----~------------~-------~--~----~

2. AndAppStore.com ready for action...

I've put the final major set of requested features in, and so it's ready 
to be hit by anyone lucky enough to be unboxing their early-arrival G1...

The site now has;
- A reminder when visited from an Android device to enable to the 
"Unknown Sources" option to allow apps to install.
- A commenting system.

If you have an app you'd like to publicise (whether it be as a full app 
or looking for testers) please feel free to register and upload it, if 
you're getting a G1 please feel free to try the site out, and thanks to 
everyone for their help with the initial testing.

Al.
http://andappstore.com/

--~--~---------~--~----~------------~-------~--~----~

3. Please convert this code 1.0version from m5.

4. About alarm clock for app.

5. ImageView not displaying at all

6. Invitation to upload software to Phoload

7. Using the browser environment