Tensorflow高性能数据优化增强工具Pipeline使用详解

这篇文章主要为大家介绍了Tensorflow高性能数据优化增强工具Pipeline使用详解，有需要的朋友可以借鉴参考下，希望能够有所帮助，祝大家多多进步，早日升职加薪

安装方法

给大家介绍一个非常好用的TensorFlow数据pipeline工具。

高性能的Tensorflow Data Pipeline，使用SOTA的增强和底层优化。

pip install tensorflow-addons==0.11.2 pip install tensorflow==2.2.0 pip install sklearn

功能

High Performance tf.data pipline
Core tensorflow support for high performance
Classification data support
Bbox data support
Keypoints data support
Segmentation data support
GridMask in core tf2.x
Mosiac Augmentation in core tf2.x
CutOut in core tf2.x
Flexible and easy configuration
Gin-config support

高级用户部分

用例1，为训练创建数据Pipeline

from pipe import Funnel from bunch import Bunch """ Create a Funnel for the Pipeline! """ # Config for Funnel config = { "batch_size": 2, "image_size": [512,512], "transformations": { "flip_left_right": None, "gridmask": None, "random_rotate":None, }, "categorical_encoding":"labelencoder" } config = Bunch(config) pipeline = Funnel(data_path="testdata", config=config, datatype="categorical") pipeline = pipeline.dataset(type="train") # Pipline ready to use, iter over it to use. # Custom loop example. for data in pipeline: image_batch , label_batch = data[0], data[1] # you can use _loss = loss(label_batch,model.predict(image_batch)) # calculate gradients on loss and optimize the model. print(image_batch,label_batch)

用例2，为验证创建数据Pipeline

from pipe import Funnel from bunch import Bunch """ Create a Funnel for the Pipeline! """ # Config for Funnel config = { "batch_size": 1, "image_size": [512,512], "transformations": { }, "categorical_encoding":"labelencoder" } config = Bunch(config) pipeline = Funnel(data_path="testdata", config=config, datatype="categorical", training=False) pipeline = pipeline.dataset(type="val") # use pipeline to validate your data on model. loss = [] for data in pipeline: image_batch , actual_label_batch = data[0], data[1] # pred_label_batch = model.predict(image_batch) # loss.append(calc_loss(actual_label_batch,pred_label_batch)) print(image_batch,label_batch)

初学者部分

Keras 兼容性

使用keras model.fit来构建非常简单的pipeline。

import tensorflow as tf from pipe import Funnel """ Create a Funnel for the Pipeline! """ config = { "batch_size": 2, "image_size": [100, 100], "transformations": { "flip_left_right": None, "gridmask": None, "random_rotate": None, }, "categorical_encoding": "labelencoder", } pipeline = Funnel(data_path="testdata", config=config, datatype="categorical") pipeline = pipeline.dataset(type="train") # Create Keras model model = tf.keras.applications.VGG16( include_top=True, weights=None,input_shape=(100,100,3), pooling=None, classes=2, classifier_activation='sigmoid' ) # compile model.compile(loss='mse', optimizer='adam') # pass pipeline as iterable model.fit(pipeline , batch_size=2,steps_per_epoch=5,verbose=1)