Class TensorRtInferenceBindings

Namespace

JYPPX.TensorRtSharp

Assembly

JYPPX.TensorRtSharp.dll

High-level TensorRT inference binding set for deployment-oriented enqueue flows. 面向模型部署 enqueue 流程的高层 TensorRT 推理绑定集。

public sealed class TensorRtInferenceBindings : IDisposable

Inheritance

object

TensorRtInferenceBindings

Implements

IDisposable

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This class owns only the CUDA buffers that it allocates itself. Buffers supplied through UseDeviceBuffer(string, CudaMemory, TensorRtDims?) remain owned by the caller. 该类只拥有自己分配的 CUDA 缓冲区；通过 UseDeviceBuffer(string, CudaMemory, TensorRtDims?) 传入的缓冲区仍由调用方拥有。

Constructors

TensorRtInferenceBindings(TensorRtEngine, TensorRtExecutionContext, int)

Creates a binding set for one engine, execution context, and optimization profile. 为一个 engine、execution context 和 optimization profile 创建绑定集。

public TensorRtInferenceBindings(TensorRtEngine engine, TensorRtExecutionContext context, int profileIndex = 0)

Parameters

engine TensorRtEngine

The TensorRT engine. TensorRT 引擎。

context TensorRtExecutionContext

The execution context used for enqueue. 用于 enqueue 的 execution context。

profileIndex int

The optimization profile index. Optimization profile 索引。

Properties

Buffers

Gets the current managed CUDA buffers keyed by TensorRT tensor name. 获取按 TensorRT tensor 名称索引的当前托管 CUDA 缓冲区。

public IReadOnlyDictionary<string, TensorRtInferenceBuffer> Buffers { get; }

Property Value

IReadOnlyDictionary<string, TensorRtInferenceBuffer>

ProfileIndex

Gets the optimization profile index used by this binding set. 获取该绑定集使用的 optimization profile 索引。

public int ProfileIndex { get; }

Property Value

int

Report

Gets the latest engine binding report. 获取最近一次 engine 绑定报告。

public TensorRtEngineBindingReport Report { get; }

Property Value

TensorRtEngineBindingReport

Methods

AllocateDeviceBuffer(string, TensorRtDims?, int?)

Allocates a CUDA buffer for the named tensor. 为指定 tensor 分配 CUDA 缓冲区。

public TensorRtInferenceBuffer AllocateDeviceBuffer(string tensorName, TensorRtDims? runtimeShape = null, int? sizeInBytes = null)

Parameters

tensorName string

The TensorRT tensor name. TensorRT tensor 名称。

runtimeShape TensorRtDims

Optional runtime shape used for size estimation. 用于估算大小的可选运行时 shape。

sizeInBytes int?

Optional explicit allocation size. 可选的显式分配字节数。

Returns

TensorRtInferenceBuffer

The created buffer descriptor. 创建的缓冲区描述。

BindAll()

Binds every attached CUDA buffer to the execution context. 将所有已附加 CUDA 缓冲区绑定到 execution context。

public TensorRtInferenceBindings BindAll()

Returns

TensorRtInferenceBindings

The current binding set for chaining. 当前绑定集，便于链式调用。

BindTensor(string)

Binds one tensor buffer to the execution context. 将一个 tensor 缓冲区绑定到 execution context。

public TensorRtInferenceBindings BindTensor(string tensorName)

Parameters

tensorName string

The TensorRT tensor name. TensorRT tensor 名称。

Returns

TensorRtInferenceBindings

The current binding set for chaining. 当前绑定集，便于链式调用。

CopyInputFromHost(string, byte[], TensorRtDims?)

Copies byte input data into a named input buffer, allocating the buffer when needed. 将字节输入数据复制到指定输入缓冲区；必要时自动分配缓冲区。

public TensorRtInferenceBindings CopyInputFromHost(string tensorName, byte[] bytes, TensorRtDims? runtimeShape = null)

Parameters

tensorName string

The input tensor name. 输入 tensor 名称。

bytes byte[]

The input bytes. 输入字节。

runtimeShape TensorRtDims

Optional runtime shape used for allocation. 用于分配的可选运行时 shape。

Returns

TensorRtInferenceBindings

The current binding set for chaining. 当前绑定集，便于链式调用。

CopyInputFromHost(string, float[], TensorRtDims?)

Copies single-precision input data into a named input buffer, allocating the buffer when needed. 将单精度输入数据复制到指定输入缓冲区；必要时自动分配缓冲区。

public TensorRtInferenceBindings CopyInputFromHost(string tensorName, float[] values, TensorRtDims? runtimeShape = null)

Parameters

tensorName string

The input tensor name. 输入 tensor 名称。

values float[]

The input values. 输入数据。

runtimeShape TensorRtDims

Optional runtime shape used for allocation. 用于分配的可选运行时 shape。

Returns

TensorRtInferenceBindings

The current binding set for chaining. 当前绑定集，便于链式调用。

Describe()

Creates a readable multi-line summary for deployment diagnostics. 创建用于部署诊断的多行可读摘要。

public string Describe()

Returns

string

A summary string. 摘要字符串。

Dispose()

Releases CUDA buffers owned by this binding set. 释放该绑定集拥有的 CUDA 缓冲区。

public void Dispose()

EnqueueAsync(CudaStream, bool, bool)

Enqueues inference asynchronously on a CUDA stream, optionally synchronizing before return. 在 CUDA stream 上异步提交推理，并可选择返回前同步。

public TensorRtInferenceExecutionSummary EnqueueAsync(CudaStream stream, bool synchronize = false, bool runShapeInference = true)

Parameters

stream CudaStream

The CUDA stream used for enqueue. 用于 enqueue 的 CUDA stream。

synchronize bool

Whether to synchronize the stream before returning. 是否在返回前同步 stream。

runShapeInference bool

Whether to run TensorRT shape inference for readiness validation. 是否为就绪校验执行 TensorRT shape inference。

Returns

TensorRtInferenceExecutionSummary

An execution summary. 执行摘要。

GetReadiness(bool)

Gets the current execution-context readiness snapshot. 获取当前 execution context 的就绪状态快照。

public TensorRtExecutionContextReadiness GetReadiness(bool runShapeInference = true)

Parameters

runShapeInference bool

Whether to run TensorRT shape inference first. 是否先执行 TensorRT shape inference。

Returns

TensorRtExecutionContextReadiness

The readiness snapshot. 就绪状态快照。

ReadOutputSingles(string, int)

Copies a named output tensor to a single-precision managed array. 将指定输出 tensor 复制到单精度托管数组。

public float[] ReadOutputSingles(string tensorName, int elementCount)

Parameters

tensorName string

The output tensor name. 输出 tensor 名称。

elementCount int

The number of float elements to read. 要读取的 float 元素数量。

Returns

float[]

The copied output values. 复制出的输出数据。

SetInputShape(string, TensorRtDims)

Sets a dynamic input shape on the execution context and refreshes the binding report. 在 execution context 上设置动态输入 shape，并刷新绑定报告。

public TensorRtInferenceBindings SetInputShape(string tensorName, TensorRtDims shape)

Parameters

tensorName string

The input tensor name. 输入 tensor 名称。

shape TensorRtDims

The runtime input shape. 运行时输入 shape。

Returns

TensorRtInferenceBindings

The current binding set for chaining. 当前绑定集，便于链式调用。

UseDeviceBuffer(string, CudaMemory, TensorRtDims?)

Attaches an externally owned CUDA buffer to the named tensor. 将外部拥有的 CUDA 缓冲区附加到指定 tensor。

public TensorRtInferenceBuffer UseDeviceBuffer(string tensorName, CudaMemory memory, TensorRtDims? runtimeShape = null)

Parameters

tensorName string

The TensorRT tensor name. TensorRT tensor 名称。

memory CudaMemory

The externally owned CUDA allocation. 外部拥有的 CUDA 设备内存。

runtimeShape TensorRtDims

Optional runtime shape used for validation and diagnostics. 用于验证和诊断的可选运行时 shape。

Returns

TensorRtInferenceBuffer

The buffer descriptor. 缓冲区描述。