René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "Datasets",
       "headline": "Datasets",
       "description": "Dataset classes in PyTorch C++ \u2014 Dataset, MapDataset, StreamDataset, and built-in datasets like MNIST.",
       "url": "/api/data/datasets.html",
       "articleBody": "Datasets# The dataset abstraction defines how to access individual samples in your data. All datasets inherit from Dataset and must implement get() and size(). Dataset Base Class# template\u003ctypename Self, typename SingleExample = Example\u003c\u003e\u003eclass Dataset : public torch::data::datasets::BatchDataset\u003cSelf, std::vector\u003cExample\u003c\u003e\u003e\u003e# A dataset that can yield data in batches, or as individual examples. A Dataset is a BatchDataset, because it supports random access and therefore batched access is implemented (by default) by calling the random access indexing function for each index in the requested batch of indices. This can be customized. Public Types using ExampleType = SingleExample# Public Functions virtual ExampleType get(size_t index) = 0# Returns the example at the given index. inline virtual std::vector\u003cExampleType\u003e get_batch(ArrayRef\u003csize_t\u003e indices) override# Returns a batch of data. The default implementation calls get() for every requested index in the batch. template\u003ctypename Self, typename Batch = std::vector\u003cExample\u003c\u003e\u003e, typename BatchRequest = ArrayRef\u003csize_t\u003e\u003eclass BatchDataset# A dataset that can yield data only in batches. Subclassed by torch::data::datasets::Dataset\u003c MNIST \u003e, torch::data::datasets::Dataset\u003c TensorDataset, TensorExample \u003e, torch::data::datasets::StatefulDataset\u003c ChunkDataset\u003c ChunkReader, samplers::RandomSampler, samplers::RandomSampler \u003e, ChunkReader::BatchType, size_t \u003e Public Types using SelfType = Self# using BatchType = Batch# using BatchRequestType = BatchRequest# Public Functions virtual ~BatchDataset() = default# virtual Batch get_batch(BatchRequest request) = 0# Returns a batch of data given an index. virtual std::optional\u003csize_t\u003e size() const = 0# Returns the size of the dataset, or an empty std::optional if it is unsized. template\u003ctypename TransformType\u003einline MapDataset\u003cSelf, TransformType\u003e map(TransformType transform) \u0026# Creates a MapDataset that applies the given transform to this dataset. template\u003ctypename TransformType\u003einline MapDataset\u003cSelf, TransformType\u003e map(TransformType transform) \u0026\u0026# Creates a MapDataset that applies the given transform to this dataset. Public Static Attributes static constexpr bool is_stateful = detail::is_optional\u003cBatchType\u003e::value# StatefulDataset# A dataset that manages its own state across batches (e.g., position in a stream). Unlike Dataset, it produces batches directly without external samplers. template\u003ctypename Self, typename Batch = std::vector\u003cExample\u003c\u003e\u003e, typename BatchRequest = size_t\u003eclass StatefulDataset : public BatchDataset\u003cSelf, std::optional\u003cstd::vector\u003cExample\u003c\u003e\u003e\u003e, size_t\u003e# A stateful dataset is a dataset that maintains some internal state, which will be reset() at the beginning of each epoch. Subclasses can override the reset() method to configure this behavior. Further, the return type of a stateful dataset\u2019s get_batch() method is always an optional. When the stateful dataset wants to indicate to the dataloader that its epoch has ended, it should return an empty optional. The dataloader knows to modify its implementation based on whether the dataset is stateless or stateful. Note that when subclassing a from StatefulDataset\u003cSelf, T\u003e, the return type of get_batch(), which the subclass must override, will be optional\u003cT\u003e (i.e. the type specified in the StatefulDataset specialization is automatically boxed into an optional for the dataset\u2019s BatchType). Public Functions virtual void reset() = 0# Resets internal state of the dataset. virtual void save(serialize::OutputArchive \u0026archive) const = 0# Saves the statefulDataset\u2019s state to OutputArchive. virtual void load(serialize::InputArchive \u0026archive) = 0# Deserializes the statefulDataset\u2019s state from the archive. ChunkDataReader# Interface for reading chunks of data from a data source. Used with ChunkDataset for large-scale data loading. template\u003ctypename ExampleType_, typename ChunkType_ = std::vector\u003cExampleType_\u003e\u003eclass ChunkDataReader# Interface for chunk reader, which performs data chunking and reading of entire chunks. A chunk could be an entire file, such as an audio data file or an image, or part of a file in the case of a large text-file split based on seek positions. Public Types using ChunkType = ChunkType_# using ExampleType = ExampleType_# Public Functions virtual ~ChunkDataReader() = default# virtual ChunkType read_chunk(size_t chunk_index) = 0# Read an entire chunk. virtual size_t chunk_count() = 0# Returns the number of chunks available in this reader. virtual void reset() = 0# This will clear any internal state associate with this reader. Custom Dataset Example# class CustomDataset : public torch::data::datasets::Dataset\u003cCustomDataset\u003e { public: explicit CustomDataset(const std::string\u0026 root) { // Load data from root directory } torch::data::Example\u003c\u003e get(size_t index) override { return {images_[index], labels_[index]}; } torch::optional\u003csize_t\u003e size() const override { return images_.size(0); } private: torch::Tensor images_, labels_; }; MapDataset# template\u003ctypename SourceDataset, typename AppliedTransform\u003eclass MapDataset : public torch::data::datasets::BatchDataset\u003cMapDataset\u003cSourceDataset, AppliedTransform\u003e, detail::optional_if_t\u003cSourceDataset::is_stateful, AppliedTransform::OutputBatchType\u003e, SourceDataset::BatchRequestType\u003e# A MapDataset is a dataset that applies a transform to a source dataset. Public Types using DatasetType = SourceDataset# using TransformType = AppliedTransform# using BatchRequestType = typename SourceDataset::BatchRequestType# using OutputBatchType = detail::optional_if_t\u003cSourceDataset::is_stateful, typename AppliedTransform::OutputBatchType\u003e# Public Functions inline MapDataset(DatasetType dataset, TransformType transform)# inline virtual OutputBatchType get_batch(BatchRequestType indices) override# Gets a batch from the source dataset and applies the transform to it, returning the result. inline virtual std::optional\u003csize_t\u003e size() const noexcept override# Returns the size of the source dataset. inline void reset()# Calls reset() on the underlying dataset. NOTE: Stateless datasets do not have a reset() method, so a call to this method will only compile for stateful datasets (which have a reset() method). inline const SourceDataset \u0026dataset() noexcept# Returns the underlying dataset. inline const AppliedTransform \u0026transform() noexcept# Returns the transform being applied. ChunkDataset# template\u003ctypename ChunkReader, typename ChunkSampler = samplers::RandomSampler, typename ExampleSampler = samplers::RandomSampler\u003eclass ChunkDataset : public torch::data::datasets::StatefulDataset\u003cChunkDataset\u003cChunkReader, samplers::RandomSampler, samplers::RandomSampler\u003e, ChunkReader::BatchType, size_t\u003e# A stateful dataset that support hierarchical sampling and prefetching of entre chunks. Unlike regular dataset, chunk dataset require two samplers to operate and keeps an internal state. ChunkSampler selects, which chunk to load next, while the ExampleSampler determines the order of Examples that are returned in each get_batch call. The hierarchical sampling approach used here is inspired by this paper http://martin.zinkevich.org/publications/nips2010.pdf Public Types using BatchType = std::optional\u003ctypename ChunkReader::BatchType\u003e# using UnwrappedBatchType = typename ChunkReader::BatchType# using BatchRequestType = size_t# using ChunkSamplerType = ChunkSampler# using ExampleSamplerType = ExampleSampler# Public Functions inline ChunkDataset(ChunkReader chunk_reader, ChunkSampler chunk_sampler, ExampleSampler example_sampler, ChunkDatasetOptions options, std::function\u003cvoid(UnwrappedBatchType\u0026)\u003e preprocessing_policy = std::function\u003cvoid(UnwrappedBatchType\u0026)\u003e())# inline ~ChunkDataset() override# inline BatchType get_batch(size_t batch_size) override# Default get_batch method of BatchDataset. This method returns Example batches created from the preloaded chunks. The implementation is dataset agnostic and does not need overriding in different chunk datasets. inline BatchType get_batch()# Helper method around get_batch as batch_size is not strictly necessary. inline virtual void reset() override# This will clear any internal state and starts the internal prefetching mechanism for the chunk dataset. inline virtual std::optional\u003csize_t\u003e size() const override# size is not used for chunk dataset. inline ChunkSamplerType \u0026chunk_sampler()# inline virtual void save(serialize::OutputArchive \u0026archive) const override# Saves the statefulDataset\u2019s state to OutputArchive. inline virtual void load(serialize::InputArchive \u0026archive) override# Deserializes the statefulDataset\u2019s state from the archive. SharedBatchDataset# template\u003ctypename UnderlyingDataset\u003eclass SharedBatchDataset : public torch::data::datasets::BatchDataset\u003cSharedBatchDataset\u003cUnderlyingDataset\u003e, UnderlyingDataset::BatchType, UnderlyingDataset::BatchRequestType\u003e# A dataset that wraps another dataset in a shared pointer and implements the BatchDataset API, delegating all calls to the shared instance. This is useful when you want all worker threads in the dataloader to access the same dataset instance. The dataset must take care of synchronization and thread-safe access itself. Use torch::data::datasets::make_shared_dataset() to create a new SharedBatchDataset like you would a std::shared_ptr. Public Types using BatchType = typename UnderlyingDataset::BatchType# using BatchRequestType = typename UnderlyingDataset::BatchRequestType# Public Functions inline SharedBatchDataset(std::shared_ptr\u003cUnderlyingDataset\u003e shared_dataset)# Constructs a new SharedBatchDataset from a shared_ptr to the UnderlyingDataset. inline virtual BatchType get_batch(BatchRequestType request) override# Calls get_batch on the underlying dataset. inline virtual std::optional\u003csize_t\u003e size() const override# Returns the size from the underlying dataset. inline UnderlyingDataset \u0026operator*()# Accesses the underlying dataset. inline const UnderlyingDataset \u0026operator*() const# Accesses the underlying dataset. inline UnderlyingDataset *operator-\u003e()# Accesses the underlying dataset. inline const UnderlyingDataset *operator-\u003e() const# Accesses the underlying dataset. inline void reset()# Calls reset() on the underlying dataset. Built-in Datasets# MNIST# class MNIST : public torch::data::datasets::Dataset\u003cMNIST\u003e# The MNIST dataset. Public Types enum class Mode# The mode in which the dataset is loaded. Values: enumerator kTrain# enumerator kTest# Public Functions explicit MNIST(const std::string \u0026root, Mode mode = Mode::kTrain)# Loads the MNIST dataset from the root path. The supplied root path should contain the content of the unzipped MNIST dataset, available from http://yann.lecun.com/exdb/mnist. virtual Example get(size_t index) override# Returns the Example at the given index. virtual std::optional\u003csize_t\u003e size() const override# Returns the size of the dataset. bool is_train() const noexcept# Returns true if this is the training subset of MNIST. const Tensor \u0026images() const# Returns all images stacked into a single tensor. const Tensor \u0026targets() const# Returns all targets stacked into a single tensor. Example: auto dataset = torch::data::datasets::MNIST(\"./data\") .map(torch::data::transforms::Normalize\u003c\u003e(0.1307, 0.3081)) .map(torch::data::transforms::Stack\u003c\u003e()); Example Struct# template\u003ctypename Data = at::Tensor, typename Target = at::Tensor\u003estruct Example# An Example from a dataset. A dataset consists of data and an associated target (label). Public Types using DataType = Data# using TargetType = Target# Public Functions Example() = default# inline Example(Data data, Target target)# Public Members Data data# Target target#",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "https://pytorch.org/docs/stable/_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/api/data/datasets.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }