From:

https://medium.com/ymedialabs-innovation/how-to-use-tfrecord-with-datasets-and-iterators-in-tensorflow-with-code-samples-ffee57d298af

import tensorflow as tf

data_arr = [
    {
        'int_data': 108,
        'float_data': 2.45,
        'str_data': 'String 100',
        'float_list_data': [256.78, 13.9]
    },
    {
        'int_data': 37,
        'float_data': 84.3,
        'str_data': 'String 200',
        'float_list_data': [1.34, 843.9, 65.22]
    }
]

def get_example_object(data_record):
    # Convert individual data into a list of int64 or float or bytes
    int_list1 = tf.train.Int64List(value = [data_record['int_data']])
    float_list1 = tf.train.FloatList(value = [data_record['float_data']])
    # Convert string data into list of bytes
    str_list1 = tf.train.BytesList(value = [data_record['str_data'].encode('utf-8')])
    float_list2 = tf.train.FloatList(value = data_record['float_list_data'])

    # Create a dictionary with above lists individually wrapped in Feature
    feature_key_value_pair = {
        'int_list1': tf.train.Feature(int64_list = int_list1),
        'float_list1': tf.train.Feature(float_list = float_list1),
        'str_list1': tf.train.Feature(bytes_list = str_list1),
        'float_list2': tf.train.Feature(float_list = float_list2)
    }

    # Create Features object with above feature dictionary
    features = tf.train.Features(feature = feature_key_value_pair)

    # Create Example object with features
    example = tf.train.Example(features = features)
    return example

with tf.python_io.TFRecordWriter('example.tfrecord') as tfwriter:
    # Iterate through all records
    for data_record in data_arr:
        example = get_example_object(data_record)

        # Append each example into tfrecord
        tfwriter.write(example.SerializeToString())

It’s like:

hashes -> dictionary -> tf.train.Feature -> tf.train.Features -> tf.train.Example -> tfrecord

Other examples are at https://github.com/Prasad9/TFRecord_Images

Images stored in string to tfrecord is like:

Similarly, images stored in bytes -> tfrecord uses this function to read img:

# Read image data in terms of bytes
        with tf.gfile.FastGFile(img_path, 'rb') as fid:
            image_data = fid.read()

For jpeg and png images, need an extra step to decode images according to it’s type:

(jpeg and png are both ok using tf.image.decode_png and tf.image.decode_jpeg correspondingly)

        # Read image data in terms of bytes
        with tf.gfile.FastGFile(img_path, 'rb') as fid:
            image_data = fid.read()

            # Encode PNG data to JPEG data
            if self._is_png_image(img_path):
                image_data = self._convert_png_to_jpeg(image_data)
        png_enc = tf.image.decode_png(self.png_img_pl, channels = 3)
        # Set how much quality of image you would like to retain while conversion
        self.png_to_jpeg = tf.image.encode_jpeg(png_enc, format = 'rgb', quality = 100)

 

gluoncv is based on mxnet, it’s very flexible and easy to use for object detection.

Since mxnet is almost ready on rocm platform, gluoncv would be possiblely available to be used on rocm.

This post is to track to building of gluoncv on rocm.

1. git clone

git clone --recurse-submodules https://github.com/ROCmSoftwarePlatform/mxnet.git -b downstream_27Mar2018

there are 3rd party components so –recurse-submodules is needed.

2. hipfft has been replaced by rocfft, so change hipfft to rocfft in mxnet/Makefile.

3. mxnet requested version is 1.3.0, but the downstream branch is 1.2.0. So changed this version in gluon-cv/gluoncv/__init__.py.

4. met an error when running:

from gluoncv import model_zoo, data, utils

net = model_zoo.get_model('ssd_512_resnet50_v1_voc', pretrained=True)

mxnet.gluon.Block has no attribute ‘load_parameters’. The current supported function is ‘load_params’. So changed it in gluon-cv/gluoncv/model_zoo/ssd/ssd.py

5. slice_like is missing in mxnet/symbol/gen_op.py.

Similar issue is reported here: https://discuss.gluon.ai/t/topic/7934/11

seems need to update mxnet. (need to check newer version of rocm based mxnet)

hacked gluoncv/model_zoo/ssd/anchor.py:

def hybrid_forward(self, F, x, anchors):
    #a = F.slice_like(anchors, x * 0, axes=(2, 3))
    #------- rocky 20181022: ---------------
    a = F.slice(anchors, begin=tuple([0]*len(anchors.shape)), end=(None,None, x.shape[2], x.shape[3], None))
    #---------------------------------------

temp solved this issue.

6. mxnet.base.MXNetError: Cannot find argument ‘valid_thresh’

There is no ‘valid_thresh’ parameter in def box_nms in:

mxnet/python/mxnet/symbol/gen_contrib.py

So remove this param from  gluoncv/model_zoo/ssd/ssd.py

        if self.nms_thresh > 0 and self.nms_thresh < 1:

            #------- rockyli, mxnet 1.2.0 has no valid_thresh param for contrib.box_nms)
            print "debug before F.contrib.box_nms"
            print result
            result = F.contrib.box_nms(
                #result, overlap_thresh=self.nms_thresh, topk=self.nms_topk, valid_thresh=0.01,
                result, overlap_thresh=self.nms_thresh, topk=self.nms_topk,
                id_index=0, score_index=1, coord_start=2, force_suppress=False)

This can temp solve above issue.

Now the object detection is Ok using a pre trained ssd VOC model with a sample image.

 

www.000webhost.com