Search for a command to run...
No comments yet. Be the first to comment.
I recently watched a short video by Carly Taylor on gamer mindset. In that 1 minute video, she talks about Chris Sawyer and how he wrote the awesome game of Roller Coaster Tycoon entirely in assembly (all by himself) because the compute power at the ...
I recently published an article on how to build an MCP server in C# (https://tjgokken.com/building-your-first-model-context-protocol-server-in-c). On that article, we built a simple MCP-style server. It remembered things, stored context in memory. Th...
Because APIs Aren't Confusing Enough Already
Or how to tame the container beast without selling your soul to YAML
Bringing It All Together: K8s Hackathon
In the .NET ecosystem, handling the conversion of the MNIST dataset from its original binary format directly within a C# application is not as straightforward as in Python due to the lack of ready-to-use libraries for this task. However, you'll see that it is not as bad as one might think.
To read the MNIST binary files in C#, you need to manually parse the binary data.
MNIST File Format: MNIST binary files for images have a specific format:
Magic number (4 bytes)
Number of images (4 bytes)
Rows (4 bytes)
Columns (4 bytes)
Pixel data (1 byte per pixel)
Read and Process the Files: We will be using the BinaryReader class in .NET to read these binary files.
Here is the code for reading the MNIST image files and converting them to a CSV file in C#:
static void ConvertToCsv(string imagePath, string labelPath, string outputCsvPath)
{
using (var imageStream = new FileStream(imagePath, FileMode.Open, FileAccess.Read))
using (var labelStream = new FileStream(labelPath, FileMode.Open, FileAccess.Read))
using (var imageReader = new BinaryReader(imageStream))
using (var labelReader = new BinaryReader(labelStream))
using (var csvWriter = new StreamWriter(outputCsvPath, false, Encoding.UTF8))
{
// Read and validate magic numbers and number of items
var imageMagicNumber = ReadInt32BigEndian(imageReader);
var numberOfImages = ReadInt32BigEndian(imageReader);
var rows = ReadInt32BigEndian(imageReader);
var cols = ReadInt32BigEndian(imageReader);
var labelMagicNumber = ReadInt32BigEndian(labelReader);
var numberOfLabels = ReadInt32BigEndian(labelReader);
if (imageMagicNumber != 2051 || labelMagicNumber != 2049)
throw new Exception("Invalid MNIST file format.");
if (numberOfImages != numberOfLabels)
throw new Exception("The number of images does not match the number of labels.");
for (var i = 0; i < numberOfImages; i++)
{
var label = labelReader.ReadByte();
var pixels = imageReader.ReadBytes(rows * cols);
var csvLine = new StringBuilder(label.ToString());
foreach (var pixel in pixels)
{
csvLine.Append($",{pixel}");
}
csvWriter.WriteLine(csvLine.ToString());
if (i % 1000 == 0)
Console.WriteLine($"Processed {i} images to CSV.");
}
}
Console.WriteLine("CSV conversion complete.");
}
The code above writes out the CSV file in chunks and optionally flushes the data to the file every 1,000 images. This helps memory usage, especially when processing large datasets.
If you want to convert to individual images instead, here is the way to do it:
static void ConvertToImages(string imagePath, string labelPath, string outputImageDir)
{
using (var imageStream = new FileStream(imagePath, FileMode.Open, FileAccess.Read))
using (var labelStream = new FileStream(labelPath, FileMode.Open, FileAccess.Read))
using (var imageReader = new BinaryReader(imageStream))
using (var labelReader = new BinaryReader(labelStream))
{
// Read and validate magic numbers and number of items
var imageMagicNumber = ReadInt32BigEndian(imageReader);
var numberOfImages = ReadInt32BigEndian(imageReader);
var rows = ReadInt32BigEndian(imageReader);
var cols = ReadInt32BigEndian(imageReader);
var labelMagicNumber = ReadInt32BigEndian(labelReader);
var numberOfLabels = ReadInt32BigEndian(labelReader);
if (imageMagicNumber != 2051 || labelMagicNumber != 2049)
throw new Exception("Invalid MNIST file format.");
if (numberOfImages != numberOfLabels)
throw new Exception("The number of images does not match the number of labels.");
Directory.CreateDirectory(outputImageDir);
for (var i = 0; i < numberOfImages; i++)
{
var label = labelReader.ReadByte();
var pixels = imageReader.ReadBytes(rows * cols);
using (var img = new Bitmap(cols, rows))
{
for (var r = 0; r < rows; r++)
{
for (var c = 0; c < cols; c++)
{
var pixelIndex = r * cols + c;
var colorValue = pixels[pixelIndex];
img.SetPixel(c, r, Color.FromArgb(colorValue, colorValue, colorValue));
}
}
img.Save(Path.Combine(outputImageDir, $"label_{label}_image_{i}.png"));
}
if (i % 1000 == 0)
Console.WriteLine($"Processed {i} images to PNG.");
}
}
Console.WriteLine("Image conversion complete.");
}
Both of the methods above make use of a method called ReadInt32BigEndian which handles the conversion from big-endian (as stored in MNIST files) to little-endian, which is the default byte order used by .NET. This is done by reversing the bytes read by BinaryReader, which reads data in little-endian format by default.
static int ReadInt32BigEndian(BinaryReader reader)
{
var data = reader.ReadBytes(4);
Array.Reverse(data);
return BitConverter.ToInt32(data, 0);
}
The .NET Console Project can be found here: https://github.com/tjgokcen/MNISTConverter-NET