Initial database population from large data files, several problems

by Justin Allen Jaynes » Fri, 06 Mar 2009 05:13:56 GMT


Sponsored Links
 I'm building a dictionary application with 135,000 word entries (words 
only).  My raw file must have been too large (1.5 meg), because I got 
this error:

D/asset (909): Data exceeds UNCOMPRESS_DATA_MAX (1424000 vs 1048576)

I've searched for this error with very few relevant hits.  It seemed to 
mean I could not open an uncompressed file over a meg.  So I then split 
the file into two smaller files and ran my code on both of them.  It 
worked out fine.  My total application size is 3 meg installed.

My code is:
public void onCreate(SQLiteDatabase database) {
    database.execSQL("CREATE TABLE " + DATABASE_TABLE + " (wordid 
INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, word VARCHAR);");
           
    Scanner fileScanner = new 
Scanner(myContext.getResources().openRawResource(R.raw.wordlist));
    while ( fileScanner.hasNextLine() ) {
        String wordFromFile = fileScanner.nextLine();
         database.execSQL("INSERT INTO words (word) VALUES ('" + 
wordFromFile + "');");
    }
    fileScanner = new 
Scanner(myContext.getResources().openRawResource(R.raw.wordlist2));
    while ( fileScanner.hasNextLine() ) {
        String wordFromFile = fileScanner.nextLine();
         database.execSQL("INSERT INTO words (word) VALUES ('" + 
wordFromFile + "');");
     }
}

However, when the application is first run, it takes several MINUTES to 
initialize the database in this way.  Is there a way (like a copy 
command, as found in, say, postgresql, or a restore of a database file) 
to copy data from a raw file, and can such a method be accessed from the 
SDK so that standard first-run procedures can correctly set up the 
database?  I have been unable to locate such a luxury.  I am seeking to 
speed up this data populating process.

First question: how can I speed up my database population?

Second question: is there a way to read a raw resource file larger than 
1 megabyte (aside from making it into two smaller files)?  If not, why?

Justin

--~--~---------~--~----~------------~-------~--~----~