Add BaseFilesystemReader interface + implement in SimpleDirectoryReader #13424
Conversation
dosubot
bot
added
the
size:L
| return self.list_files(**kwargs) | ||
|
|
||
| @abstractmethod | ||
| def get_file_info(self, input_file: Path, **kwargs) -> Dict: |
There was a problem hiding this comment.
Am wondering if this should return a FileInfo/FileMetadata pydantic class which just contains some minimum required metadata fields to standardize this a bit 🤔
| def read_file(self, input_file: Path, **kwargs) -> List[Document]: | ||
| """Read file from filesystem and return documents.""" |
There was a problem hiding this comment.
should this return the file content instead? to separate the file parsing from file loading
…rface-simpledirreader
| @@ -71,6 +71,141 @@ class Config: | |||
| arbitrary_types_allowed = True | |||
|
|
|||
|
|
|||
| class BaseResourcesReader(BaseReader): | |||
There was a problem hiding this comment.
This looks good to me, and probably fine to merge for now.
But in general, nesting base class under base class can be a confusing pattern to maintain 😅 Longer term I'd prefer just a single base class, or mixin's that add functionality
Because now we have BaseReader -> BaseResourcesReader -> BaseFileSystemReader ->
| @@ -43,7 +43,7 @@ name = "llama-index-core" | |||
| packages = [{include = "llama_index"}] | |||
| readme = "README.md" | |||
| repository = "https://github.com/run-llama/llama_index" | |||
| version = "0.10.37" | |||
| version = "0.10.37.post1" | |||
There was a problem hiding this comment.
fyi this would need to be manually published after merging (happy to do that)
| from llama_index.core.async_utils import run_jobs, get_asyncio_module | ||
| from llama_index.core.schema import Document | ||
| from tqdm import tqdm | ||
|
|
||
|
|
||
| class BaseFilesystemReader(BaseResourcesReader, ABC): | ||
| @abstractmethod | ||
| def read_file_content(self, input_file: Path, **kwargs) -> bytes: |
There was a problem hiding this comment.
Kind of curious how this differs from load_resource() -- feels like it would be the same method?
There was a problem hiding this comment.
I guess this returns bytes
…rface-simpledirreader
Description
This is just the changes made to
llama-index-corein #13408. Opened a separate PR for these because the changes done to integrations there (like S3 and SharePoint) depend on the core package version being bumped first to pass CI.#13408 also contains the unit tests for the implementation in SimpleDirectoryReader
Version Bump?
Type of Change