Bulk operations

Bulk operations are helpful when you need to create or update several documents within one Database with a low amount of requests. In particular, the _bulk_docs endpoint allows to write a bunch of documents in one request.

Bulk operations in aiocouch are similar to transactions. You define the set of affected Document, apply the changes and finally perform the bulk request. Depending on the particular task, you need to use one of two context manager classes.

For example, the following code affects the documents foo and baz, existing or not, and sets the key llama to awesome with one bulk request.

async with database.update_docs(["foo", "baz"], create=True) as bulk:
    async for doc in bulk:
        doc["llama"] = "awesome"

Include documents in bulk operations

Affected documents can be defined in two ways. The first way is to pass a list of document ids as the ids parameter.

async with database.update_docs(ids=["foo", "baz"]) as bulk:
    ...

The second method is the usage of the append() method. Just pass an instance of Document and its content will be saved as part of the bulk operation.

the_document = Document(...)

async with BulkOperation(database=my_database) as bulk:
    bulk.append(the_document)

Once the control flow leaves the context, the bulk operation persists the applied changes to all documents that there included in the bulk operation one or the other way. Also, both ways can be mixed.

Create many documents in one operation

To create many documents, you use the create_docs() method to get the context manager. Include documents as described above. Once the context manager closes, one request containing all document contents gets send to the server.

async with my_database.create_docs(...) as bulk:
    for doc in bulk:
        # make changes to the Document instances

# the request was send now

Note that the bulk operation does not check, if the requested documents alrady exists on the server. Instead, the error list will contain conflict in the error field corresponding to the document.

Update many documents in one operation

To update many documents, you use the update_docs() method to get the context manager. Include documents as described above. Once the context manager closes, one request containing all document contents gets send to the server. In contrast to the create operation, the BulkUpdateOperation context manager will request all documents whose ids where passed as the ids parameter. If you already have Document instance, you may want to use the append() method instead.

my_doc: Document = ...

async with my_database.update_docs(...) as bulk:
    bulk.append(my_doc)

    for doc in docs:
        # make changes to the Document instances

# the request was send now

Error handling for bulk operations

The important bit first, none of the bulk operation context manager will raise an exception if something in the request went wrong. Each individual document can be saved successfully or may have an error. It’s in your responsibility to check the status after the request finished.

You can check the status of each document with the ok, error, and response properties of the context manager. The ok and error lists contain all documents that could and couldn’t be saved properly, respectively. The response contains the response from the CouchDB server. So in case of an error, it will contain a description of what went wrong.

async with BulkOperation(database=my_database) as bulk:
    ...

if len(bulk.error) == 0:
    print(f"Saved all {len(bulk.ok)} documents")
else:
    print(f"Failed to saved {len(bulk.error)} documents")

Reference

class aiocouch.bulk.BulkOperation(database)

A context manager for bulk operation. This operation allows to write many documents in one request.

Bulk operations use the _bulk_docs endpoint of the database.

To populate the list of written documents, use the append() method.

Parameters

database (Database) – The database used in the bulk operation

async for ... in __aiter__()

An iterator that yields Document instances that are part of this bulk operation.

Returns

Every Document instance that will be affected by this operation

Return type

AsyncGenerator[Document, None]

append(doc)

Add a document to this bulk operation.

Parameters

doc (Document) – the document that should be stored as part of the bulk operation

Raises

ValueError – if the provided document instance is already part of the bulk operation

Return type

Document

Returns

the provided document

create(id, data=None)

Create a new document as part of the bulk operation

Parameters

• id (str) – the id of the document

• data (Optional[Dict[str, Any]]) – the inital data used to set the body of the document, defaults to None

Raises

ValueError – if the provided document id is already part of the bulk operation

Return type

Document

Returns

a Document instance reference the newly created document

error: Optional[List[Document]]

The list of all Document instances that could not be saved to server.

Only available after the context manager has finished without a passing exception.

ok: Optional[List[Document]]

The list of all Document instances that there successfully saved to server.

Only available after the context manager has finished without a passing exception.

response: Optional[List[Dict[str, Any]]]

The resulting JSON response of the _bulk_docs request. Refer to the CouchDB documentation for the contents.

Only available after the context manager has finished without a passing exception.

property status: Optional[List[Dict[str, Any]]]

Deprecated since version 2.1.0: Use the response property instead.

update(doc)

Add a document to this bulk operation.

Parameters

doc (Document) – the document that should be stored as part of the bulk operation

Raises

ValueError – if the provided document instance is already part of the bulk operation

Return type

Document

Returns

the provided document

Deprecated since version 2.1.0: Use append(doc) instead. It just makes more sense.

class aiocouch.bulk.BulkCreateOperation(database, ids=[])

A context manager for bulk creation operations. This operation allows to write many documents in one request.

Bulk operations use the _bulk_docs endpoint of the database.

Parameters

• database (Database) – The database used in the bulk operation

• ids (List[str]) – a list of ids of the involved documents, defaults to []

error: Optional[List[Document]]

The list of all Document instances that could not be saved to server.

Only available after the context manager has finished without a passing exception.

ok: Optional[List[Document]]

The list of all Document instances that there successfully saved to server.

Only available after the context manager has finished without a passing exception.

response: Optional[List[Dict[str, Any]]]

The resulting JSON response of the _bulk_docs request. Refer to the CouchDB documentation for the contents.

Only available after the context manager has finished without a passing exception.

class aiocouch.bulk.BulkUpdateOperation(database, ids=[], create=False)

A context manager for bulk update of documents. In particular, for every provided id, a Document instance is provided. The data is fetched using the AllDocsView with a minimal amount of requests.

Parameters

• database (Database) – The database of the bulk operation

• ids (List[str]) – list of document ids

• create (bool) – If True, every document contained in ids that doesn’t exist, will be represented by an empty Document instance.

error: Optional[List[Document]]

The list of all Document instances that could not be saved to server.

Only available after the context manager has finished without a passing exception.

ok: Optional[List[Document]]

The list of all Document instances that there successfully saved to server.

Only available after the context manager has finished without a passing exception.

response: Optional[List[Dict[str, Any]]]

The resulting JSON response of the _bulk_docs request. Refer to the CouchDB documentation for the contents.

Only available after the context manager has finished without a passing exception.