[Catalyst] Re: Large requests with JSON?
moseley at hank.org
Tue Feb 9 15:12:21 GMT 2010
On Tue, Feb 9, 2010 at 2:36 AM, Aristotle Pagaltzis <pagaltzis at gmx.de>wrote:
> > 3) upload a document for the user and the document must include
> > an associated collection of meta data (e.g. filename,
> > timestamp, author etc.). The uploaded document must include
> > this meta data before it can be accepted.
> That sounds like the case I was thinking about: just do a PUT
> request with X-MyApp-Filename, X-MyApp-Timestamp etc headers.
Of course, I left out the ability to upload multiple flies at once. Doing
that with headers could get ugly. (X-MyApp-Filename-01,
X-MyApp-Filename-02, ...) Of course, could just not provide that
multiple-file upload ability to API users and limit it to web users. That
would work ok.
With XML-RPC we just have multiple <upload> struct elements that are
containers for the meta data and the base64 file contents.
> (Another option, which is better in some ways I think, would be
> the two-request approach as above, though that would be more
> complicated. Ie. the client POSTs the metadata, the server files
> the data away temporarily and returns a link to which the client
> can PUT the file, and only once that request has succeeded does
> the server store both metadata and file in their proper place.)
That's a bit of redesign of the application for a two-phase upload. Seems a
shame to have to add new database tables and cron jobs to clean up
incomplete uploads just because of my choice of serialization. I agree
that's probably the cleanest design, though. From past experience I can
assume some customers will have trouble adding request headers for the
libraries they are using.
form-data is possible serialization, but it's a flat serialization so also
need to have fields like filename_01, title_01, filename_02, title_02 to
handle multiple uploads at once. (Plus, the app already handles that
form-data). I'm not sure how much meta data can be associated with an
upload in form-data (other than filename, content-disposition, and
content-type), or if the libraries clients use to create a request can be
XML-RPC is ugly but nicely handles multiple uploads with associated meta
data for each, and can be stream parsed so that the base64 file data is
chunked to a temp file and not stored in memory.
JSON provides the nice nested structures but, IIUC, has to be in-memory to
parse. I hate those "out of memory!" messages, so it would be very nice to
not have the file uploads in JSON.
Not pretty at all, but maybe using form-data with a JSON-encoded "meta"
field that has a list of uploads with associated meta-data including a
field_name with each upload that associates it with a field that contained
the uploaded file. Most client libraries have a way to send form-data, so
that would be easy for customers to implement.
None of those are great options.
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Catalyst