OutOfMemoryError: how best to transfer large video files into a byte array?

by Anna PS » Fri, 09 Apr 2010 04:59:02 GMT


Sponsored Links
 Apologies for cross-posting with StackOverflow, but I'm getting a bit
desperate. I'll cross-post any final answer too.

Please could anyone suggest an approach for transferring a >2MB video
from a ContentResolver into a Bytestream, without running out of
memory?

See question: 
 http://stackoverflow.com/questions/2599305/android-outofmemoryerror-while-uploading-video-how-best-to-chunk 

Here's the current code, which throws an OutOfMemoryError on the
byteBuffer.write(buffer, 0, len) line when transferring large videos:

// get bytestream to upload
videoByteArray = getBytesFromFile(cR, fileUriString);

public static byte[] getBytesFromFile(ContentResolver cR, String
fileUriString) throws IOException {
    Uri tempuri = Uri.parse(fileUriString);
    InputStream is = cR.openInputStream(tempuri);
    byte[] b3 = readBytes(is);
    is.close();
    return b3;
}
public static byte[] readBytes(InputStream inputStream) throws
IOException {
    ByteArrayOutputStream byteBuffer = new ByteArrayOutputStream();
    // this is storage overwritten on each iteration with bytes
    int bufferSize = 1024;
    byte[] buffer = new byte[bufferSize];
    int len = 0;
    while ((len = inputStream.read(buffer)) != -1) {
        byteBuffer.write(buffer, 0, len);
    }
    return byteBuffer.toByteArray();
}

--



OutOfMemoryError: how best to transfer large video files into a byte array?

by Abdul Mateen » Fri, 09 Apr 2010 05:03:17 GMT


 It tried very much on this, but I had to gave up, it seems not possible as
far as I think, lets get the experts reply.






>


Sponsored Links


OutOfMemoryError: how best to transfer large video files into a byte array?

by Mark Murphy » Fri, 09 Apr 2010 05:16:35 GMT


 


1. Why are you using a ContentProvider?

2. What are you doing with the >2MB byte array when you get it on the
ContentResolver side?

My hope is that there's a way to avoid the problem altogether.

-- 
Mark Murphy (a Commons Guy)
 http://commonsware.com  |  http://twitter.com/commonsguy 

Android App Developer Books:  http://commonsware.com/books 

--



OutOfMemoryError: how best to transfer large video files into a byte array?

by Anna PS » Fri, 09 Apr 2010 05:23:18 GMT


 > 1. Why are you using a ContentProvider?

Because that seemed to be the best way to get from a content URI
(returned from the Android video camera intent) to a byte stream.
Happy to do it another way, though.


Once I've got the videoByteArray, I'm uploading it to a server, as a
FilePart within a multipart message. I can post the code for that if
it would be useful.

It works well for files of less than 2MB.


That would be good :) If anyone can work it out, it's probably you!

--



OutOfMemoryError: how best to transfer large video files into a byte array?

by Mark Murphy » Fri, 09 Apr 2010 05:35:22 GMT


 


Oh. I thought you had perhaps written the ContentProvider in question.


I suspect your problem is not in reading the InputStream, but rather in
how FilePart does what it does. Since that isn't part of Android, it was
probably developed without tight memory constraints in mind. If you hand
it 1 KB chunks each time, it is probably going to allocate a 1 KB byte
array, then a 2 KB byte array, then a 3 KB byte array, then a 4 KB byte
array, and so on. You are hopelessly fragmenting your heap. Not to
mention that FilePart might well wind up Base64 encoding the result,
requiring another massive memory allocation.

I would recommend rewriting your Web server. Use HTTP PUT instead of
HTTP POST with multipart. Better yet, use HTTP PUT and support chunking
on the Web server -- you do a PUT for every 1MB or so and the server
stitches them together.

If this is not your Web server, you're pretty much screwed AFAICT. Doing
bigger chunks than 1 KB will help somewhat, and rewriting FilePart to
minimize reallocations will help somewhat, but you'll still probably not
get much past, say, 5MB that way.

-- 
Mark Murphy (a Commons Guy)
 http://commonsware.com  |  http://twitter.com/commonsguy 

Android Training in NYC: 30 April-2 May 2010:  http://guruloft.com 

--



OutOfMemoryError: how best to transfer large video files into a byte array?

by Bob Kerns » Fri, 09 Apr 2010 07:19:46 GMT


 I'm a bit puzzled why the other respondents didn't pick up this, maybe
I missed something in the discussion?

But anyway, the basic flaw in your strategy is reading it all into
memory at once. There's no need to do so, and that is ALWAYS going to
impose a limit on the size of file you can handle, on any platform.

Instead of reading all your data, and then stufffng it into a
ByteArrayOutputStream, and pulling out a *HUGE* array --

The simplest strategy: Open an output stream to your server, and
simply copy from the camera to the server.

That has the fatal drawback that the camera will be tied up, and
possibly blocked, if the server is too slow -- which is likely. So
that's not really a viable strategy. So you need something to stand in
for the server. The only viable option is a file on the SD card.

So open a temporary output file, rather than a byte-array output
stream, and write it there. Then, instead of getting a byte array, re-
open your SD card file, and copy from that file to the server.

Under no circumstances, in no programing environment, do you ever want
to buffer a file of unknown size in memory.

If you wanted to be extremely clever, you could simultaneously
transfer to the SD card, and transfer from the SD card to the net. But
extreme cleverness is seldom very smart.

I wonder why the multipart stuff was left out of the SDK? I hadn't
noticed that. These little holes and omissions and using-obsolete-
versions thing I find rather disorienting -- you think you know what's
there, then BAM -- not on the Android, unless you BYO Code.



>



OutOfMemoryError: how best to transfer large video files into a byte array?

by Mark Murphy » Fri, 09 Apr 2010 07:41:01 GMT


 


I didn't miss it. OTOH, I was trying to scope my response into something
that didn't mess up the original plan too much and stuck to things I
knew worked with HttpClient. And, as previously noted, I'm short on
sleep. :-(


I'm reasonably certain the Uri being handed back from the Intent will be
to a file handled by the MediaStore, not live off the camera.


It's an oft-asked question with no known answer.

-- 
Mark Murphy (a Commons Guy)
 http://commonsware.com  |  http://twitter.com/commonsguy 

Android 2.x Programming Books:  http://commonsware.com/books 

--



OutOfMemoryError: how best to transfer large video files into a byte array?

by Gubatron » Sat, 10 Apr 2010 01:20:05 GMT


 I suppose when you say FilePart, you mean this

org.apache.commons.httpclient.methods.multipart.FilePart

I'm thinking along these lines after looking at that API (I haven't
tested this)

final File theFile = new File("yourFileLargerThan2Mb.ext");

//Implement your own PartSource to feed your File
PartSource  partSource = new PartSource() {
   //this should return a buffered reader... right?
   public InputStream createInputStream() {
      return BufferedInputStream (new FileInputStream(theFile));
   }

   //implement the other methods of the interface
   public String getFileName() {
     return theFile.getName();
   }

   public long getLength() {
     return theFile.length();// although this might be how much is
left on the stream, not sure.
   }

}

FilePart part = FilePart(theFileName, partSource);

then use your part on your multipart request.

On Apr 9, 12:43pm, Anna Powell-Smith <annapowellsm...@googlemail.com>



--



OutOfMemoryError: how best to transfer large video files into a byte array?

by Matt Kanninen » Thu, 22 Apr 2010 03:04:44 GMT


  think you are on the right track.  I have had very similar problems
in the past. For me the temporary file solution was obvious because I
had to modify the media before I uploaded it. Once you have a
temporary file, then you have the generic problem of uploading a large
file to a web server, and you aren't using don't need any Android
specific code.

It also makes it much easier if there are interstitial pages, multiple
activities client side that will need to access the temporary media
file, (perhaps to display a preview), and you can handle low memory
conditions better. The more Android development I do the more I find
myself writing to private local storage, to make sure I never lose
state.

On Apr 21, 9:02am, Android Development <indodr...@gmail.com> wrote:
> >



Other Threads

1. can get the music information?

hi all,

is there are api that can get the music information,
such as album name,author name,song name or time of this song?

thanks
--~--~---------~--~----~------------~-------~--~----~

2. javax.xml.datatype.XMLGregorianCalendar

An external library that I use (Sesame) references the method
javax.xml.datatype.XMLGregorianCalendar.toXMLFormat, which is not
available in Android. What can I do about that? The complete error is
this:

E/ActivityThread( 1811): Failed to find provider info for
com.google.settings
W/dalvikvm( 1811): VFY: unable to find class referenced in signature
(Ljavax/xml/datatype/XMLGregorianCalendar;)
E/dalvikvm( 1811): Could not find method
javax.xml.datatype.XMLGregorianCalendar.toXMLFormat, referenced from
method org.openrdf.model.impl.ValueFactoryBase.createLiteral
W/dalvikvm( 1811): VFY: unable to resolve virtual method 2047: Ljavax/
xml/datatype/XMLGregorianCalendar;.toXMLFormat ()Ljava/lang/String;
W/dalvikvm( 1811): VFY:  rejecting opcode 0x6e at 0x0000
W/dalvikvm( 1811): VFY:  rejected Lorg/openrdf/model/impl/
ValueFactoryBase;.createLiteral (Ljavax/xml/datatype/
XMLGregorianCalendar;)Lorg/openrdf/model/Literal;
W/dalvikvm( 1811): Verifier rejected class Lorg/openrdf/model/impl/
ValueFactoryBase;
D/AndroidRuntime( 1811): Shutting down VM
W/dalvikvm( 1811): threadid=3: thread exiting with uncaught exception
(group=0x4000fe70)
E/AndroidRuntime( 1811): Uncaught handler: thread main exiting due to
uncaught exception
E/AndroidRuntime( 1811): java.lang.VerifyError:
org.openrdf.model.impl.ValueFactoryBase
E/AndroidRuntime( 1811):        at org.openrdf.http.client.HTTPClient.<init>
(HTTPClient.java:133)
E/AndroidRuntime( 1811):        at
org.openrdf.repository.http.HTTPRepository.<init>(HTTPRepository.java:
56)
E/AndroidRuntime( 1811):        at
org.openrdf.repository.http.HTTPRepository.<init>(HTTPRepository.java:
61)
E/AndroidRuntime( 1811):        at com.test.TestActivity.onCreate
(TestActivity.java:27)
E/AndroidRuntime( 1811):        at
android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:
1123)
E/AndroidRuntime( 1811):        at
android.app.ActivityThread.performLaunchActivity(ActivityThread.java:
2231)
E/AndroidRuntime( 1811):        at
android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:
2284)
E/AndroidRuntime( 1811):        at android.app.ActivityThread.access$1800
(ActivityThread.java:112)
E/AndroidRuntime( 1811):        at android.app.ActivityThread$H.handleMessage
(ActivityThread.java:1692)
E/AndroidRuntime( 1811):        at android.os.Handler.dispatchMessage
(Handler.java:99)
E/AndroidRuntime( 1811):        at android.os.Looper.loop(Looper.java:123)
E/AndroidRuntime( 1811):        at android.app.ActivityThread.main
(ActivityThread.java:3948)
E/AndroidRuntime( 1811):        at java.lang.reflect.Method.invokeNative
(Native Method)
E/AndroidRuntime( 1811):        at java.lang.reflect.Method.invoke
(Method.java:521)
E/AndroidRuntime( 1811):        at com.android.internal.os.ZygoteInit
$MethodAndArgsCaller.run(ZygoteInit.java:782)
E/AndroidRuntime( 1811):        at com.android.internal.os.ZygoteInit.main
(ZygoteInit.java:540)
E/AndroidRuntime( 1811):        at dalvik.system.NativeStart.main(Native
Method)
--~--~---------~--~----~------------~-------~--~----~

3. display more than one activity window on device

4. Making a persistant content observer

5. App Test Group - Progress

6. screen Y position not changing after soft keyboard pushes window up.

7. Android/Davlik as Real Time OS?