[Catalyst] Re: Large requests with JSON?

[Bill Moseley]

On Tue, Feb 9, 2010 at 2:36 AM, Aristotle Pagaltzis <pagaltzis at gmx.de>wrote:

> > 3) upload a document for the user and the document must include
> > an associated collection of meta data (e.g. filename,
> > timestamp, author etc.). The uploaded document must include
> > this meta data before it can be accepted.
>
> That sounds like the case I was thinking about: just do a PUT
> request with X-MyApp-Filename, X-MyApp-Timestamp etc headers.
>

Of course, I left out the ability to upload multiple flies at once.  Doing
that with headers could get ugly.  (X-MyApp-Filename-01,
X-MyApp-Filename-02, ...)  Of course, could just not provide that
multiple-file upload ability to API users and limit it to web users.  That
would work ok.

With XML-RPC we just have multiple <upload> struct elements that are
containers for the meta data and the base64 file contents.



> (Another option, which is better in some ways I think, would be
> the two-request approach as above, though that would be more
> complicated. Ie. the client POSTs the metadata, the server files
> the data away temporarily and returns a link to which the client
> can PUT the file, and only once that request has succeeded does
> the server store both metadata and file in their proper place.)
>

That's a bit of redesign of the application for a two-phase upload.  Seems a
shame to have to add new database tables and cron jobs to clean up
incomplete uploads just because of my choice of serialization.  I agree
that's probably the cleanest design, though.  From past experience I can
assume some customers will have trouble adding request headers for the
libraries they are using.

form-data is possible serialization, but it's a flat serialization so also
need to have fields like filename_01, title_01, filename_02, title_02  to
handle multiple uploads at once.  (Plus, the app already handles that
form-data).  I'm not sure how much meta data can be associated with an
upload in form-data (other than filename, content-disposition, and
content-type), or if the libraries clients use to create a request can be
that creative.

XML-RPC is ugly but nicely handles multiple uploads with associated meta
data for each, and can be stream parsed so that the base64 file data is
chunked to a temp file and not stored in memory.

JSON provides the nice nested structures but, IIUC, has to be in-memory to
parse.  I hate those "out of memory!" messages, so it would be very nice to
not have the file uploads in JSON.

Not pretty at all, but maybe using form-data with a JSON-encoded "meta"
field that has a list of uploads with associated meta-data including a
field_name with each upload that associates it with a field that contained
the uploaded file.  Most client libraries have a way to send form-data, so
that would be easy for customers to implement.

None of those are great options.


-- =

Bill Moseley
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20100209/05abe=
26b/attachment.htm