基于FPGA的车牌识别系统

基于FPGA的车牌识别系统

这个项目是之前参加"FPGA创新设计邀请赛"时开始写的程序,后来参加学校"罗姆杯"的时候又改进了一下.程序基于Xilinx公司的Pynq-Z2开发板,使用opencv库完成车牌识别.

项目背景和设计目的

       车牌识别系统是计算机视频图像识别技术在车辆牌照识别中的一种应用,在高速公路、停车场、小区、道路等环境下有着广泛的应用。

       我们希望能够充分利用PYNQ的内部资源,运用Python语言的程序设计和OpenCV计算机视觉库,设计出一个较为可靠的车牌识别系统,将输出结果显示到显示器上,包含车牌号码和车速等信息。

       对于停车场门口或收费站等应用场景,本系统还可以直接控制舵机,用于限制车辆的进出

 

产品特点

       运用了Python语言的程序设计和OpenCV计算机视觉库,使用了ZYNQ-7020芯片,将ARM处理器嵌入到FPGA中,这样将ARM处理器的优势和FPGA的优势结合到了一起。

       研究基于FPGA+ARM平台的车牌识别系统可利用ARM处理器在操作系统方面的优势,能够实现良好的人机交互功能;又利用FPGA在并行算法加速、可动态重配置的特点,实现硬件加速、增加系统的灵活性,提高车牌识别速度。

       使用USB外设单目摄像头读取图像信息,能准确、清晰地搜集到车辆图像,图像分辨率达到设计要求。使用外设显示屏输出显示车牌信息。

       外接舵机,模拟小区出入口和高速公路收费站。

 

硬件平台

1.png

    使用Xilinx公司的PYNQ-Z2开发板,搭载ZYNQ-7020芯片和ARM-A9双核处理,使用Python语言和Verilog HDL硬件描述语言对板子进行描述。

2.png

性能和成本

现场可编程门阵列(FPGA)和图形处理器(GPU)的性能对比。FPGA的功耗远小于GPU,并且FPGA可以根据特定的应用去编程硬件,虽然FPGA的运算速度相较GPU较慢,总体上,谁在机器视觉方面具备优势是一个需要深入研究的课题,众多企业对FPGA的应用前景比较看好,具备较强的研究价值和应用前景。

FPGA的造价相对GPU较低,在未来,随着技术的成熟和广泛的应用,FPGA的制造成本和销售价格推测均会进一步降低,在现在和将来能够带来的经济收益会高于GPU,在这方面,FPGA应用于机器视觉领域具备很强的潜在的社会经济效益。

 

整体框图

3.png

FPGA处理模块

4.png

算法流程图

5.png

运行结果

6-.png

7.png

9.png

 

改进

图像处理的拓展。在本次设计中,尚未加入车型识别/驾驶员特征(如有无系安全带等)等内容的读取,在进一步研究中可以考虑增加上述内容的研究和识别。

加速模块尚未完善,加速效果不显著。

物联网方向的深入。在本次设计中,尚未设置联网比对功能,如果有这部分的设计,那么我们可以实时与公安的数据进行比对,如车辆颜色等特征判断是否为套牌,车牌是否系伪造(数据库中无该车牌),是否是被盗车辆、未缴纳交通强制保险车辆、肇事逃逸车辆等,功能会更加完善、齐全。

 

Python代码

    以下为python代码,但是并不包含KNN模型的数据,完整的工程请看:

https://github.com/chenjianqu/Pynq-LicensePlateRecognition


from pynq.overlays.base import BaseOverlay
from pynq.lib.video import *
from matplotlib import pyplot as plt
import numpy as np
import cv2
import time
import math
from time import sleep
from pynq.lib.pmod import Pmod_PWM
from pynq.lib.pmod import PMODA
 
 
base = BaseOverlay("base.bit")
pwm = Pmod_PWM(PMODA, 0)
 
#读取字典和数据       
with open('knn_dict_letter.txt','r') as f:
    labelDict = eval(f.read())
with np.load('knn_data.npz') as data:
    train_data=data['train_data']
    train_labels=data['train_labels']
with np.load('knn_data_zh.npz') as data:
    train_data_zh=data['train_data_zh']
    train_labels_zh=data['train_labels_zh']
 
    # 定义分类器
cascade_path = 'cascade.xml'
cascade_car_path = 'cars.xml'
car_cascade = cv2.CascadeClassifier(cascade_car_path)
cascade = cv2.CascadeClassifier(cascade_path)
 
 
# monitor configuration: 640*480 @ 24Hz
Mode = VideoMode(640,480,24)
hdmi_out = base.video.hdmi_out
hdmi_out.configure(Mode,PIXEL_BGR)
hdmi_out.start()
 
# monitor (output) frame buffer size
frame_out_w = 640
frame_out_h = 480
 
scale=400/480 #比例系数
 
PLATE_WIDTH=0.44
 
Hmin=100
Hmax=124
Smin=120
Smax=255
Vmin=120
Vmax=255
 
 
reg=["ShangHaiF19911","ShangHaiB6N866","NingXia1B6N86","ShangHaiB6N866","ShangHaiB8N866","ShangHaiB0N866","NingXiaB19873","HeNanP8A629"]
 
 
isRecognize=3
 
print('Read Data Done')
 
def KeyCallBack():
    if(base.buttons[0].read()==1):
        isRecognize=0
    elif(base.buttons[1].read()==1):
        isRecognize=1
    elif(base.buttons[2].read()==1):
        isRecognize=2
    elif(base.buttons[3].read()==1):
        isRecognize=3
 
#精确的定位车牌
def locateAccuracy(img):
    frame=cv2.cvtColor(img,cv2.COLOR_BGR2HSV)
    height = frame.shape[0]
    width = frame.shape[1]
    top=0
    button=height-1
    left=0
    right=width-1
 
    row=0
    while(row<height):
        col=0
        count=0
        while(col<width):
            if((frame[row,col,0]>=Hmin and frame[row,col,0]<=Hmax) and (frame[row,col,1]>=Smin and frame[row,col,1]<=Smax) and (frame[row,col,2]>=Vmin and frame[row,col,2]<=Vmax)):
                count+=1
            col+=1
        if(count/width>0.6):
            top=row
            break
        row+=1
 
    row=button
    while(row>0):
        col=0
        count=0
        while(col<width):
            if((frame[row,col,0]>=Hmin and frame[row,col,0]<=Hmax) and (frame[row,col,1]>=Smin and frame[row,col,1]<=Smax) and (frame[row,col,2]>=Vmin and frame[row,col,2]<=Vmax)):
                count+=1
            col+=1
        if(count/width>0.6):
            button=row
            break
        row-=1
 
    col=right
    while(col>0):
        row=0
        count=0
        while(row<height):
            if((frame[row,col,0]>=Hmin and frame[row,col,0]<=Hmax) and (frame[row,col,1]>=Smin and frame[row,col,1]<=Smax) and (frame[row,col,2]>=Vmin and frame[row,col,2]<=Vmax)):
                count+=1
            row+=1
        if(count/height>0.6):
            right=col
            break
        col-=1
    col=left
    while(col<width):
        row=0
        count=0
        while(row<height):
            if((frame[row,col,0]>=Hmin and frame[row,col,0]<=Hmax) and (frame[row,col,1]>=Smin and frame[row,col,1]<=Smax) and (frame[row,col,2]>=Vmin and frame[row,col,2]<=Vmax)):
                count+=1
            row+=1
        if(count/height>0.6):
            left=col
            break
        col+=1
    return top,button,left,right
 
def recognize(img):
    top,button,left,right=locateAccuracy(img)
    image=img[top:button,left:right]
    if(image.size==0):
        return []
    img_gray=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
    img_thre=cv2.adaptiveThreshold(img_gray,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,35,2)#自适应二值化
    #对对图像进行垂直投影
    arr=np.zeros(image.shape[1])
    col=0
    while(col<img_thre.shape[1]):
        row=0
        count=0
        while(row<img_thre.shape[0]):
            count+=img_thre[row,col]
            row+=1
        arr[col]=int(count/255)
        col+=1
    #根据投影结果进行分割字符
    count_1=0
    flag=0
    flag_index=0
    i=0
    for c in arr:
        if(c<10):
            count_1+=1
        else:
            if(count_1>flag):
                flag=count_1
                flag_index=int(i-count_1/2)
            if(count_1>3):
                arr[int(i-count_1/2)]=-1
            count_1=0
        i+=1
    i=0
    j=0
    x=0
    y=top
    h=button-top
 
    #获得分割结果
    charList=[]
    for c in arr:
        if(c==-1):
            w=i-x
            charList.append([x+left,y,w,h])
            x=i
        if(flag_index==c and (j!=1 or j!=2)):
            return []
 
        i+=1
        j+=1
    charList.append([x+left,y,right-x,h])
    if(len(charList)<=5 or len(charList)>8):
        return []
    return charList
 
def recognizeRect(img_thre,rect,knn):
    x,y,w,h=rect
    roi=img_thre[y:(y+h),x:(x+w)]
    if h>w:
        roi=cv2.copyMakeBorder(roi,0,0,int((h-w)/2),int((h-w)/2),cv2.BORDER_CONSTANT,value=[0,0,0])
    roi=cv2.resize(roi,(20,20))
    #cv2.imshow("char",roi)
    #cv2.waitKey(0)
    roivector=roi.reshape(1,400).astype(np.float32)
    ret,result,neighbours,dist=knn.findNearest(roivector,k=6)#进行预测
    return int(ret)
 
def access_pixels(frame):
    frame = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)
    #print(frame.shape)  #shape内包含三个元素:按顺序为高、宽、通道数
    height = frame.shape[0]
    weight = frame.shape[1]
    count=0
    for row in range(height):            #遍历高
        for col in range(weight):         #遍历宽
            if((frame[row,col,0]>=100 and frame[row,col,0]<=124) and (frame[row,col,1]>=43 and frame[row,col,1]<=255) and (frame[row,col,2]>=46 and frame[row,col,2]<=255)):
                count+=1
    if(count/(height*weight)>0.5):
        return True
    else:
        return False
 
def isPlate(frame):
    if(frame.shape[1]>frame.shape[0]/2 or frame.shape[1]*5<frame.shape[0]):
        return True
    else:
        return False
 
 
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640);
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480);
 
print("Capture device is open: " + str(cap.isOpened()))
 
period=20000
duty=6
pwm.generate(period,duty)
 
time_start=time.time()
time_last=0
i=1
x_last=0
y_last=0
isRecognize=3
str_plate=""
print("start while")
while(cap.isOpened()):   
    if(base.buttons[0].read()==1):
        isRecognize=0
    elif(base.buttons[1].read()==1):
        isRecognize=1
    elif(base.buttons[2].read()==1):
        isRecognize=2
 
    elif(base.buttons[3].read()==1):
        isRecognize=3
        period=20000
        duty=6
        pwm.generate(period,duty)
        str_plate=""
 
 
    time_start=time.time()
    ret, frame = cap.read()
 
    if (ret):  
        if(isRecognize==3):
            outframe = hdmi_out.newframe()
            outframe[0:640,0:480,:] = frame[0:640,0:480,:]
            hdmi_out.writeframe(outframe)
        else:
            time_last=time_start
            time_start=time.time()
            #print("read frame")
            image=cv2.resize(frame,(int(640*scale),400))
            #print("after new outframe time is:"+str(time.time()-time_start))
            img_gray=cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
            #print("after cvtgray time is:"+str(time.time()-time_start))
            car_plates = cascade.detectMultiScale(img_gray, 1.1, 2, minSize=(36, 9), maxSize=(36 * 40, 9 * 40))
            #print("after mul time is:"+str(time.time()-time_start))
 
 
            for car_plate in car_plates:
                x, y, w, h = car_plate
                #plate = image[y: y + h, x: x + w]
                #if(isPlate(plate)==False):#根据颜色判断是否是正确的车牌区域
                #    continue
                #if(access_pixels(plate)==False):#根据颜色判断是否是正确的车牌区域
                #    continue
 
                if(isRecognize==0):
                    plateScale=(w/PLATE_WIDTH)
                    v=(math.sqrt((x-x_last)*(x-x_last)+(y-y_last)*(y-y_last))/(time_start-time_last)/plateScale)/i
                    v=int(v*100)/100
                    x_last=x
                    y_last=y
                    i=0
                    cv2.putText(image,"Vehicle Velocity:"+str(v)+" m/s",(20,image.shape[0]-10),cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 0, 255), 2)
                    i+=1
                elif(isRecognize==1):
                    if(base.buttons[0].read()==1):
                        isRecognize=0
                    elif(base.buttons[1].read()==1):
                        isRecognize=1
                    elif(base.buttons[2].read()==1):
                        isRecognize=2
                       
                    elif(base.buttons[3].read()==1):
                        isRecognize=3
                        period=20000
                        duty=6
                        pwm.generate(period,duty)
                        str_plate=""
 
                    #分割字符
                    img_thre=cv2.adaptiveThreshold(img_gray,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,35,2)#自适应二值化
                    plate=image[y: y + h, x: x + w]
                    charRect=recognize(image[y: y + h, x: x + w])
                    if(len(charRect)==0):
                        continue
                    if(len(charRect)>7):
                        charRect.pop()
                   
                    str_plate=""
                    KeyCallBack();
                    #模型创建
                    knn_zh=cv2.ml.KNearest_create()
                    knn_zh.train(train_data_zh,cv2.ml.ROW_SAMPLE,train_labels_zh)
                   
                    #识别中文
                    rect=charRect[0]
                    x1,y1,w1,h1=rect
                    x1=x+x1
                    y1=y+y1
                    str_plate=labelDict[recognizeRect(img_thre,(x1,y1,w1,h1),knn_zh)]
                    if(len(charRect)>0):
                        x1,y1,w1,h1=charRect[0]
                        cv2.rectangle(image,(x1,y1),(x1+w1,y1+h1),(255,0,0),1)
                    if(base.buttons[0].read()==1):
                        isRecognize=0
                    elif(base.buttons[1].read()==1):
                        isRecognize=1
                    elif(base.buttons[2].read()==1):
                        isRecognize=2
                       
                    elif(base.buttons[3].read()==1):
                        isRecognize=3
                        str_plate=""
                        period=20000
                        duty=6
                        pwm.generate(period,duty)
                       
 
                    knn=cv2.ml.KNearest_create()
                    knn.train(train_data,cv2.ml.ROW_SAMPLE,train_labels)
                    for rect in charRect[1:]:
                        x1,y1,w1,h1=rect
                        x1=x+x1
                        y1=y+y1
                        cv2.rectangle(image,(x1,y1),(x1+w1,y1+h1),(255,0,0),1)#框出字块
                        s=labelDict[recognizeRect(img_thre,(x1,y1,w1,h1),knn)]
                        str_plate=str_plate+s
                   
                    for re in reg:
                        if(re in str_plate):
                            period=20000
                            duty=11
                            pwm.generate(period,duty)
                            print("********************ok")
                       
                    x1,y1,w1,h1=rect
                    print('recognize time:',time.time()-time_start,'s')
                    image=cv2.putText(image,str_plate,(20,image.shape[0]-10),cv2.FONT_HERSHEY_PLAIN, 2.0, (0, 0, 255), 2)
                elif(isRecognize==2):
                    image=cv2.putText(image,str_plate,(20,image.shape[0]-10),cv2.FONT_HERSHEY_PLAIN, 2.0, (0, 0, 255), 2)
                #标出粗定位的车牌
                cv2.rectangle(image, (x - 10, y - 10), (x + w + 10, y + h + 10), (255, 255, 255), 1)
            image=cv2.resize(image,(640,480))   
            outframe = hdmi_out.newframe()
            outframe[0:640,0:480,:] = image[0:640,0:480,:]
            hdmi_out.writeframe(outframe)
            #print("after write time is:"+str(time.time()-time_start))
            
    else:
        raise RuntimeError("Failed to read from camera.")
print("end program")


首页 所有文章 机器人 计算机视觉 自然语言处理 机器学习 编程随笔 关于