o AgQ,@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZeeZ Gdd d eZ!Gd d d eZ"dS) z:Pebblo's safe dataloader is a wrapper for document loadersN)version)AnyDictIterableIteratorListOptional)Document) BaseLoader) BATCH_SIZE_BYTESPLUGIN_VERSIONApp FrameworkIndexedDocumentPebbloLoaderAPIWrappergenerate_size_based_batches get_full_pathget_loader_full_pathget_loader_type get_runtimeget_source_sizec@seZdZUdZdZeed<     d+dddded ed ed ed e ed ede ededefddZ de e fddZ d,ddZdee fddZed,ddZdefddZde efddZd ede e fd!d"Zde e fd#d$Zd%e d&ede fd'd(Zd eddfd)d*ZdS)-PebbloSafeLoaderzkPebblo Safe Loader class is a wrapper around document loaders enabling the data to be scrutinized. F_discover_sentNlocal)classifier_locationanonymize_snippetslangchain_loadernameowner descriptionapi_key load_semanticclassifier_urlrrc Cs|rt|ts td||_tt|_||_tj dp||_ ||_ ||_ t|j|_g|_g|_tt|jdddd} t| |_t|j|_t|_| |j|jd|jdkredt|jini|_||_t|||| d |_|j|jdS) NzMust specify a valid name.PEBBLO_LOAD_SEMANTIC.'r)loader source_path source_typesource_path_size)r!rr#r) isinstancestr NameErrorapp_nameuuiduuid4load_idr(osenvirongetr"rr rr)docs docs_with_idtypesplitrr*rr+r batch_sizeloader_details_get_app_detailsappr pb_clientsend_loader_discover) selfrrrr r!r"r#rr loader_namerBr/var/www/html/development/chatbot/venv/lib/python3.10/site-packages/langchain_community/document_loaders/pebblo.py__init__%s>  "    zPebbloSafeLoader.__init__returncCs|j|_||jS)zxLoad Documents. Returns: list: Documents fetched from load method of the wrapped `loader`. )r(loadr6classify_in_batches)r@rBrBrCrFVs zPebbloSafeLoader.loadc Cst|j|j}g}t|}t|D]6\}}||dk}||_||_|jj|j|j |j |d}| ||j r>| |}n|}||q||_dS)z Classify documents in batches. This is to avoid API timeouts when sending large number of documents. Batches are generated based on the page_content size. ) loading_endN)rr6r:len enumerate _index_docsr7r>classify_documentsr=r;_add_pebblo_specific_metadatar"_add_semantic_to_docs _unindex_docsextend) r@batchesprocessed_docs total_batchesibatch is_last_batchclassified_docsbatch_processed_docsrBrBrCrGas*      z$PebbloSafeLoader.classify_in_batchesc csz|j}Wnty&}z|jjjd}t|t||d}~ww zt|}Wn ty;g|_ YdSwt |f|_ | |_ |j |j |j|j}|||jra|||_ n||_ |j dVq()zLoad documents in lazy fashion. Raises: NotImplementedError: raised when lazy_load id not implemented within wrapped loader. Yields: list: Documents from loader's lazy loading. z does not implement lazy_load()NTr)r( lazy_loadNotImplementedError __class____name__loggererrornext StopIterationr6listrLr7r>rMr=r;rNr"rOrP)r@ doc_iteratorexcerr_strdocclassified_docrBrBrCrZs6           zPebbloSafeLoader.lazy_loadcCs d|_dS)NT)r)clsrBrBrCset_discover_sents z"PebbloSafeLoader.set_discover_sentc Cs:t\}}t|j|j|j|j||ttdtddd}|S)z\Fetch app details. Internal method. Returns: App: App details. langchain_community)rr)rrr r2runtime frameworkplugin_versionclient_version) rr r/rr r2r rr)r@rlrkr=rBrBrCr<s  z!PebbloSafeLoader._get_app_detailscCddt|jD}|S)z Indexes the documents and returns a list of IndexedDocument objects. Returns: List[IndexedDocument]: A list of IndexedDocument objects with unique IDs. cSs*g|]\}}tddt|i|qS)pb_idrB)rr-dict.0rUrfrBrBrC sz0PebbloSafeLoader._index_docs..)rKr6)r@r7rBrBrCrLzPebbloSafeLoader._index_docsrXcCsVdd|jD}|D]}|d}||vr||||q dd|D}|S)aF Adds semantic metadata to the given list of documents. Args: classified_docs (Dict): A dictionary of dictionaries containing the classified documents with pb_id as key. Returns: List[Document]: A list of Document objects with added semantic metadata. cSs i|] }|jt|j|jdqS) page_contentmetadata)rpr rwrxrsrfrBrBrC sz:PebbloSafeLoader._add_semantic_to_docs..rpcSsg|]}|qSrBrBryrBrBrCrtsz:PebbloSafeLoader._add_semantic_to_docs..)r7valuesr5_add_semantic_to_doc)r@rX indexed_docsrgdoc_idsemantic_metadata_docsrBrBrCrOs   z&PebbloSafeLoader._add_semantic_to_docscCro)z Converts a list of IndexedDocument objects to a list of Document objects. Returns: List[Document]: A list of Document objects. cSs g|] \}}t|j|jdqSrv)r rwrxrrrBrBrCrtsz2PebbloSafeLoader._unindex_docs..)rKr7)r@r6rBrBrCrPruzPebbloSafeLoader._unindex_docsrfrgcCs8t|di|jd<t|di|jd<|S)a4 Adds semantic metadata to the given document in-place. Args: doc (Document): A Document object. classified_doc (dict): A dictionary containing the classified document. Returns: Document: The Document object with added semantic metadata. entitiespebblo_semantic_entitiestopicspebblo_semantic_topics)rbr5keysrx)r@rfrgrBrBrCr|s   z%PebbloSafeLoader._add_semantic_to_docc Cst|jD]4}|j}|jjjdkrt|d|j|d<nt|d|d|j|d<||jidd|d<qdS)z*Add Pebblo specific metadata to documents.SharePointLoadersource full_path pb_checksumN) r7rxr(r\r]rr5r)rp)r@rXrf doc_metadatarBrBrCrNs    z.PebbloSafeLoader._add_pebblo_specific_metadata)rrNFN)rEN)r] __module__ __qualname____doc__rbool__annotations__r r-rrDrr rFrGrrZ classmethodrir r<rrLrrOrPrqr|rNrBrBrBrCrsR     1  "   rc@seZdZdZddddddeedeedeeedeeee fdeeeee fd df d d Z d e e fd d Z d ee fddZdS)PebbloTextLoaderz Loader for text data. Since PebbloSafeLoader is a wrapper around document loaders, this loader is used to load text data directly into Documents. N)ridsrx metadatastextsrrrxrrEcCs"||_||_||_||_||_dS)a Args: texts: Iterable of text data. source: Source of the text data. Optional. Defaults to None. ids: List of unique identifiers for each text. Optional. Defaults to None. metadata: Metadata for all texts. Optional. Defaults to None. metadatas: List of metadata for each text. Optional. Defaults to None. N)rrrrxr)r@rrrrxrrBrBrCrDs  zPebbloTextLoader.__init__ccst|jD]9\}}d}|jpi}|jr(|t|jkr(|j|r(||j||jr7|t|jkr7|j|}t|||dVqdS)zi Lazy load text data into Documents. Returns: Iterator of Documents N)idrwrx)rKrrxrrJupdaterr )r@rUtext_idrxrBrBrCrZ9s  zPebbloTextLoader.lazy_loadcCs g}|D]}||q|S)z` Load text data into Documents. Returns: List of Documents )rZappend)r@ documentsrfrBrBrCrFIs  zPebbloTextLoader.load)r]rrrrr-rrrrrDrr rZrFrBrBrBrCrs*   r)#rloggingr3r0importlib.metadatartypingrrrrrrlangchain_core.documentsr )langchain_community.document_loaders.baser $langchain_community.utilities.pebblor r r rrrrrrrrr getLoggerr]r^rrrBrBrBrCs    8 y